1. Field of the Invention
The present invention relates generally to computer systems and, more particularly, to methods and apparatus for determining when a processor has failed.
2. Description of the Related Art
About two decades ago, a relatively compact and basic computing device, which would come to be known as the personal computer or PC, was being developed. Like all personal computers since, these early personal computers utilized microprocessors coupled to various types of memory devices. However, due to the extremely limited computing capabilities of these early microprocessors and the limited size and costliness of high speed memory, these early personal computers truly were nothing but stand alone personal computing devices.
In the intervening years, microprocessors, memory devices, software, and many other portions of a computing system have seen rapid improvements in speed, capacity, complexity, and performance. By way of example, the latest generation microprocessors from Intel Corporation include the Pentium, Pentium Pro, and Slot-2 microprocessors. These processors are so powerful that they not only would have been considered an unbelievable evolution over the Z80 and 8080 microprocessors of two decades ago, but they also offer significant enhancements over the prior generation 486 processors. Even in view of this rapid and incredible improvement of microprocessors, the resource requirements of software are always increasing, as are the variety of uses for xe2x80x9cpersonalxe2x80x9d computers. These needs, in turn, drive the need for the design and development of ever more powerful and efficient computer systems.
In view of these vast technological improvements, personal computers have made great strides from their humble beginnings to provide solutions for the ever expanding needs and desires of the computing public. For example, two decades ago, virtually all large or complicated computing operations, from data processing to telephone networks, were handled by large mainframe computers. However, networks of microprocessor-based personal computers have made tremendous inroads into areas that were once the exclusive domain of such large mainframe computers. Such networks of personal computers provide the computing power and centralized access to data of mainframe systems, along with the distributed computing capability of stand alone personal computers. These networks typically include tens, hundreds, or even thousands of personal computers, including powerful personal computers that can act as servers. Indeed, as such networks have become larger and more complex, there has been a need for improving the computing performance of servers on the network. To address this need, multiple processors are now being used in personal computers which are configured to act as servers in order to produce more powerful servers.
The expansion of microprocessor-based personal computers into the mainframe domain, however, has not been problem free. Mainframe computers have historically been designed to be reliable and extremely fault tolerant. In other words, a failure of a portion of the mainframe computer does not typically result in lost or corrupted data or extensive down time. Because personal computer networks are increasingly being used instead of mainframe systems, users are demanding that such networks provide fault tolerance similar to that found in the mainframe systems.
In view of these user demands, manufacturers have devised various ways for providing fault tolerance in personal computer networks. Many of these developments have concentrated on the fault tolerance of the servers in a personal computer network, because servers are typically the cornerstone of most networks. In other words, because the servers typically provide applications, data, and communications among the various workstations, the failure of one server could cause the entire network to fail.
In a multiprocessor computer such as those typically used as servers, one problem that may occur involves the failure of one of the multiple processors. Because of this possibility, a fault-tolerant scheme should include the ability to detect when one of the multiple processors has failed. Current fault detection schemes of this type typically attempt to determine whether a processor has failed during the power up sequence. For example, one method of booting a multiprocessor computer involves the assignment of a primary processor which is responsible for activating the remainder of the computer system. Once the primary processor has been successfully started, the primary processor then tests the remaining processors and various other components in the computer system. While this scheme facilitates the detection of a failed secondary microprocessor, it does not address a situation where the primary microprocessor fails. In such a situation, the primary microprocessor would be unable to activate the secondary processors, leaving the entire server incapacitated even though one or more secondary processors may remain fully operational.
In an effort to address this problem, one technique utilizes a timer associated with each processor, along with specialized hardware to determine the hierarchy of the multiple processors. When the system is reset, the primary processor is initialized by the hardware and activated to boot the remainder of the computer system including the secondary processors. However, if the primary processor does not take certain actions within the period set by the timer, the timer expires and sends a signal to the hardware to cause the hierarchy of the multiple processors to be changed. Thus, one of the secondary processors becomes the primary processor, and it attempts to activate the computer system. This process continues until a successful boot operation occurs.
This technique has been revised to cause the primary processor to check an error log to determine if certain critical errors have previously occurred on that processor. If so, the booting sequence stops, and the timer causes the hardware to reestablish the hierarchy of the multiple processors. In addition, the primary processor also checks for critical errors prior to actually loading the operating system. If any errors have occurred, the primary processor transfers control to one of the secondary processors and shuts itself off.
It should be appreciated that fault detection schemes such as those described above are only capable of detecting a failed processor during the power up sequence when the server is turned on. Although this type of scheme may be quite satisfactory in certain circumstances, depending upon the frequency of power up sequences and the criticality of the network, several shortcomings still exist. For example, in many networks, the servers are only turned on and off when the network malfunctions or for routine maintenance. Clearly, if a processor fails and causes the server to be shut down, the network will at least lose the availability of that server even though the server may still contain several operable processors, and, of course, the removal of the server may cause the entire network to go down.
When a processor fails, it would be desirable to detect which processor has failed immediately so that the computer may discontinue use of the processor and so that the failed processor may be replaced. Presently used testing methods require each processor to perform an operation simultaneously. The results of the operation are compared with one another. If the results are the same, then all processors are assumed to be working properly. However, if the results do not match, then one of the processors is assumed to have failed. In a computer system having three or more processors, the failure of a single processor is relatively easy to pinpoint using this methodology. However, in a dual processor system, such a method merely provides an indication that one of the two processors has failed, without providing an indication of which particular processor has failed. Without being able to determine which processor has failed, the computer system cannot discontinue use of the failed processor and it cannot identify the failed processor for subsequent replacement.
The present invention may address one or more of the problems set forth above.
Certain aspects commensurate in scope with the originally claimed invention are set forth below. It should be understood that these aspects are presented merely to provide the reader with a brief summary of certain forms the invention might take and that these aspects are not intended to limit the scope of the invention. Indeed, the invention may encompass a variety of aspects that may not be set forth below.
In accordance with one aspect of the present invention, there is provided a method of determining a processor failure in a multiprocessor computer. The method comprising the acts of: (a) identifying each processor in the computer; (b) assigning a unique write location to each identified processor; (c) directing each processor to perform a given test and to store a result of the test in the respective unique write location; (d) comparing the result stored in each unique write location to a respective expected result; and (e) indicating a failure of a particular one of the identified processors in response to the stored result being different than the expected result. The method may also comprise the acts of: setting a rate for directing each processor to perform a given test during normal operation of the computer; and clearing each write location in response to the stored result being the same as the expected result.
In accordance with another aspect of the present invention, there is provided an apparatus for detecting a failed processor of a multiprocessor computer. The apparatus includes: means for identifying each processor in the computer; means for assigning a unique write location to each identified processor; means for directing each processor to perform a given test and to store a result of the test in the respective unique write location; means for comparing the result stored in each unique write location to a respective expected result; and means for indicating a failure of a particular one of the identified processors in response to the stored result being different than the expected result. The apparatus may also include means for setting a rate for directing each processor to perform a given test during normal operation of the computer, and means for clearing each write location in response to the stored result being the same as the expected result.
In accordance with still another aspect of the present invention, there is provided a tangible medium containing a software program for detecting a failed processor of a multiprocessor computer, which includes a routine for assigning a unique write location to each processor in the computer; a routine for directing each processor to perform a given test and to store a result of the test in the respective unique write location; a routine for comparing the result stored in each unique write location to a respective expected result; and a routine for indicating a failure of a particular one of the identified processors in response to the stored result being different than the expected result.
In accordance with yet another aspect of the present invention, there is provided a network that includes a plurality of interconnected computers, wherein at least one of the plurality of interconnected computers includes a multiprocessor computer. The multiprocessor computer includes means for detecting a failed processor of a multiprocessor computer. The detecting means includes means for identifying each processor in the multiprocessor computer; means for assigning a unique write location to each identified processor; means for directing each processor to perform a given test and to store a result of the test in the respective unique write location; means for comparing the result stored in each unique write location to a respective expected result; and means for indicating a failure of a particular one of the identified processors in response to the stored result being different than the expected result. The detecting means may also include means for setting a rate for directing each processor to perform a given test during normal operation of the computer; and means for clearing each write location in response to the stored result being the same as the expected result.
In accordance with a further aspect of the present invention, there is provided a computer that includes a plurality of microprocessors, where each microprocessor repeatedly performs a given test during operation of the computer. The computer also includes a plurality of registers, where each register is assigned to a selected one of the plurality of microprocessors for storing a result of each given test performed by the respective microprocessor. A comparator is coupled to the plurality of registers to receive the result stored in each register. The comparator compares each stored result with an expected result and generates an signal indicative of a failed microprocessor in response to one of the stored results being different than a respective one of the expected results.